Image-based Environment Matting (Online ID 213)

Environment matting is a powerful technique for modelling the complex light-transport properties of real-world optically active elements: transparent, refractive and reflective objects. Zongker et al [1999] and Chuang et al [2000] show how environment mattes can be computed for real objects under carefully controlled laboratory conditions. However, for many objects of interest, such calibration is difficult to arrange. For example, we might wish to determine the distortion caused by filming through an ancient window where the glass has flowed; we may have access only to archive footage; or we might simply want a more convenient means of acquiring the matte. We show in this sketch that accurate environment mattes can be computed from natural images, without the need for specialized calibration of the acquisition. The goal is to take a set of example images, containing the optical element of interest (e.g. the lens in figure 1), and transfer the element’s environment matte to a new background image. The technique is best understood by working backwards from the final composite of a novel background image N and the computed environment matte. Each pixel in the output collects light from a blend of pixels in N . Let us call the set of pixels which contribute to a given output pixel p the footprint of p, or p’s receptive field. Previous researchers have defined the footprint using rectangular regions [Zongker et al. 1999] or mixtures of Gaussians [Chuang et al. 2000]. In this work, we must deal with complex multimodal distributions, so we use a discrete map of source pixels, where each source pixel has an associated weight. The value of the output pixel is then computed as a weighted sum over the pixels of N . Thus if we can compute the receptive field for each pixel, we can compute the composite. In order to compute the receptive field of a given pixel p, we need at least two images: one containing the test object (e.g. the lens in figure 1), and one containing only the background. We note that pixels in the background which have contributed to p’s colour will have similar colour to p. In fact, for each background pixel, the similarity between its colour and the query colour is a function of the amount that background pixel contributes. Thus, we can obtain a bound on p’s receptive field by computing the correlation between a small (e.g. 3 × 3) window around p and each location in the background image. Such a bound is illustrated in figure 1b. Of course, for a single image, this bound is very weak—many pixels which accidentally share p’s colour are included in the receptive field. However, with a sequence of images, as in figure 1, the receptive field is constant as the background moves, and with each new image, the footprint can be refined. Figure 1d shows the refined receptive field for the indicated foreground pixel after 8 views have been integrated. Note how the single peak corresponds to the true source pixel, indicated in subimage 2. Computing the background image may be achieved by mosaicing the moving-background sequence [Irani et al. 1994] or moving the camera. Figure 2 shows an example where the camera is moved to obtain a clean view of the background. In this example, there is just one reference view, so regularizing constraints were employed in order to permit a solution: the receptive fields were assumed small and close to their source pixels. The examples show that, although its performance is scenedependent, the technique can work well given sufficiently rich